Contains hardcoded config values in infohandler, mock data inside the JS. Very much a prototype.

Walked back playbooks to focus on detections.

Made detections more data driven. Added form validation when creating a detection. TODO: form validation when editing.

Added Duplicate functionality to the API/UI, added Bulk enable/disable to API.

Touched the surface on how detections will sync.

Enabling/Disabling rules that use flowbits is different than rules that don't.

Parsing needs testing, currently failing because the rule I'm testing with uses angled quotes instead of straight quotes.

Can now delete a Detection from the UI.

UI now performs some light validation before saving a Detection.

Properly adds SIDs to disabled file.

MinLength now set in UI to match server requirement of 5.

After creating a detection, you're taken to the edit page for that detection instead of being left at a filled in create page.

If creating fails, a banner is shown.

Case insensitive sidExtractor. The extractSID function now returns nil if more than 1 SID is specified.

Tests for pure functions.

Added MemConfigStore to facilitate testing.

Test rule parsing, enabling/disabling of rules, and various helper functions.

Moved Casestore functions relating to Detections to a brand new Detectionstore.

TODO: TestSyncSuricata needs more tests, it currently only covers the bare minimum. Also, more testing needs to take place around what happens when the detections module is disabled.

Converted repeated strings to fixtures.

Fleshed out the rest of SyncSuricata's tests.

Added validation tests.

Removed permission checks left over from creating ElasticDetectionstore from ElasticCasestore.

Jerry rigged the config to always have a `suricataengine` module section so I don't have to fight salt.

Expanded on detection severity types.

Expanded the DetectionEngine interface to support how we're importing community rules.

First pass at a SyncCommunityRules.

Solved some issues around ParseSuricataRule and whitespace.

Removed the sync endpoint from detections handler. Instead of so-rule-update attempting to upload the file or trigger a sync, the SuricataEngine module will watch files on disks for changes and respond to them. The DetectionEngine interface was cleaned up in response to this change.

All sync logic was moved the SuricataEngine module. The module now maintains a long-lived goroutine that checks a configurable file at a configurable interval for changes and applies them when seen. A fingerprint of the rules file is saved in a configurable location.

TODO: Tests.

The context that the server initializes is being put to use and slightly modified. As I find new permissions I need, I'm adding roles to cover them. When the dust settles, I'll re-evaluate.

Search criteria now determine what permissions they need to check the user account for. Hints can be given but the search criteria's index and kind are tested to see if alternate permissions need to be checked.

detection/read and detection/write were quickly added to the rbac/permissions file. The roles they're attached to has not been finalized and will almost certainly change.

Lengthened the allowed title as many community rules have titles longer than 100 chars.

When RoundTrip'ing with ElasticTransport, do not add the es-security-runas-user header when the server's agent is in the ctx.

Added 2 new test rules to TestValidate inspired by community rules that couldn't parse but should've.

Successfully imported 30,000+ community rules!

Uses elasticsearch's `search_after` to page through results when the max documents of -1 is passed in.

Added logic to disallow editing of community rules. Adjusted detection's form validation rules so that community rules aren't validated at the form level.

Removed "Details" Detection tab and reordered the controls under it to the "Summary" and "Signature" tabs. Rearranged some tabs.

Cleaned up the strings used in dropdowns (engine, severity) with capitalization. Cleaner UI, but the correct casing is still passed over the wire.

Refactored PublicID's "Generate" button to be "Extract" so that it fills the field using the SID in the rule. May eventually make this field readonly except for this modification.

Updated infohandler's response to use the same severity values the rules use.

Converted more strings to i18n references.

`indexDocument` and `deleteDocument` on the ElasticEventstore now pass the ctx in. This removes the WARN log message about making a request without a user. Needs more thorough testing to be sure there's no unwanted side effects.

New library: go-git. We use it to clone/pull the sigma repo.

Prevent community rules from being updated through the detection CRUD endpoints.

Improved change detection in suricata's community importer. If a community rule's content hasn't changed then it isn't updated, significantly improving the time it takes to update community rules.

Improved support for sort criteria when querying elasticsearch.

Linting.

ElastAlert community rules are imported from zips in SigmaHQ's release page on github. The configured packages are downloaded, parsed, and fingerprinted. Currently the sigma converter isn't completely implemented until some networking issues are fixed.

GetAllCommunitySIDs can now filter by engine (or no filter if nil is passed).

Updated detection severities to match our pickiest engine so far: elastalert.

Added ID field to SigmaRules. Although it's not required, all community rules have one. This is the corresponding field to a detection's PublicID.

TODO: tests

Backend:

Removed go-github. Instead, we're using a configurable url template.

Updated config values to impact how the UI renders.

Replaced all references of github.com/tj/assert with references to github.com/stretchr/testify/assert for consistency.

Fixed an issue where errors resulted in more errors when syncing local detections in elastalert.

ElastAlert's SyncLocalDetections now implemented.

ElastAlert's syncCommunityDetections is further implemented.

Fixed a bug where syncing community detections resulted in purging local detections.

Tests. Refactor to make some functions more testable.

UI:

Hide charts when on detections page in advanced mode.

Removed auto-grow from rule text area. For large rules, modifying the rule would adjust the text area size which would adjust the browser's scroll position sometimes with counter productive effects.

Include js-yaml library to parse yaml (i.e. sigma) to retrieve values. Used to extract the PublicID field from a sigma rule. More uses to come.

`make([]any, x, x)` only needs to specify len, cap will match.

`x == nil || len(x) == 0` only the len check is necessary. len(nil) is 0.

`time.Now().Sub(...)` should be `time.Since(...)`.

Define, init, return simplified to return.

Unlike other languages, go doesn't require `break` statements at the end of cases in a switch.

SeverityTranslations. A map used to translate one engine's severity levels to the levels we support in SOC's detections.

A lot of UI work around CRUDing overrides in the Tuning tab of a detection.

A little cleanup in hunt.js. Removed helper functions from the early days of detections. Could/should have been removed a long time ago.

Initial logic for applying overrides when syncing detections started, but the modified threshold file isn't updated yet until I revisit and test everything.

Overrides can now be enabled/disabled.

When saving a detection, add the saving overlay so multiple edits can't take place at once.

Apply CustomFilter overrides when wrapping.

Save threshold modifications to disk after syncing local detections and their overrides in suricata.

Expanding one override would expand all the rows. Fixed.

Expanded CIDR checking to include IPv6.

Changed OverrideParameters yaml tags: genId => gen_id, thresholdType => type, and no yaml output for CustomFilter.

Track is a field for both Suppress and Threshold type overrides but it contains different enum values depending on which of those two types, basically Threshold doesn't have `by_either`.

When saving a detection, update the UpdatedAt field for an override when the old detection doesn't have an override with the exact same values regardless of order.

ElastAlert now enables first seen detections from the community. The infohandler now returns more predetermined search queries.

Also updated new chi references to chi/v5

Refactored the ElastAlertEngine to use a new interface for interacting with external resources (Disk and Network). This allows for mocking these resources in the new tests.

Also updated Suricata's tests to include the new thresholding config value used by overrides.

To thoroughly test the parser, the process loads all the stored rules from the yara rules folder but does not attempt to sync them to elasticsearch. Also includes tests with hand crafted queries with "features" found in the community rules.

Currently the only error this should run across is a legitimate error in a source rule: https://github.com/Security-Onion-Solutions/securityonion-yara/blob/master/yara/cn_pentestset_webshells.yar#L744

Detections can now be selected for bulk actions. The Select All checkbox toggles between selecting the entire current page and clearing all selection. If the current page is selected, a prompt to select ALL of the results becomes available. Currently the only actions that can be taken are the enabling or disabling of selected detections.

If picking individual IDs, then those IDs are sent up to the API to be modified. If ALL are selected, then the query is sent and detections matching the query will be modified.

A few component tests were added to hunt.test.js testing the SelectAll logic. Cypress tests will be added later.

UpdateDetectionField has been updated to update a detection field (i.e. without having the entire object) faster. Also should no longer be limited to one field although the functionality is not tested or implemented.

The ElasticDetectionstore now gets an ElasticSearch client passed into it so actions don't have to be run against Eventstore.

An error was fixed in the suricata parser. Better care is taken around modifying settings files that may be empty.

Settings' validator that used to check for Jinja now checks for opening AND closing jinja tags to prevent accidental false positives such as the suricata rule that contained `{%` in a URL.

When hunting, the daterange for detections will always be from the unix epoch to now. This helps us treat detections like a traditional model instead of a timescale DB entry.

Submitting a bulk operation against detections now results in a 202 (Accepted) and the work is done in a goroutine. At the end, the results are broadcast back to the client.

…th SigmaCLI

There's some side effects because the default `yaraRulesFolder` is still being used to store non-compiled rules by salt.

Updated tests around the new ExecCommand added to IOManagers.

This also includes ElastAlert switching to using the sigma cli instead of the sigconverter.io container. All references to the container have been removed.

Fixed a bug where 2 slashes before ending a string resulted in the parser not recognizing that the string terminated.

Updated "Parentheses in Unquoted Option" test to also test this case.

Moved hardcoded config values to the config in securityonion.

Previous versions of a Detection's history are now presented in the History tab. With Refresh, Sort, Search, and Details.

Updated the Detection Overview tab.

Fixed an issue where some strelka metadata was coming through quoted. Added tests for new approach.

Next step: add some missing data to the model, conversion process, and validation process so we can finish the Overview tab.

Detections now have tags, a ruleset, and a language.

Suricata's ruleset is looked up from the config and should either be ETOPEN, ETPRO, or emerging_rules.

Strelka's ruleset is named after the repo the rule was downloaded from.

ElastAlert's ruleset is the zip pkg we downloaded the rule from (core, core+, all_rules, etc).

Queries, converters, and tests have been updated to include the new fields.

TODO: Change how detections are created to ask for language (and tags) instead of engine. Also show language instead of engine in search result columns.

New Operations and Details panels implemented in Detections view.

Small consistency change to how YaraRule's check if they already have an ID before generating one.

Hidden tags. Hard wrap on detection logic. Suricata classtype for Summary, fallback to title. Reference links get a protocol if they're missing one. "None" when no references. "Signature" => "Detection Logic" on Overview tab. Remove classtype from Detection Logic. "Enable" => "Status". Duplicate and Delete buttons under the Status. Delete is disabled if looking at community rule. "Detection Type" => "Type". Public Id added above SOC Id. Use "Id" over "ID". Author changes for suricata rules. Bottom metadata spacing fixed.

TODO: License, rule-specific dates, buttons on "Detection Source" tab.

All references now have their protocols fixed instead of just suricata.

Finished up the right hand column of the detections view by including License and several pieces of data extracted from the rule.

Updated the backend to add licenses in places we know them. Except for Yara, community rules are tagged with appropriate licenses as they're imported.

Detections no longer have a Note field. Instead they have comments that work like case comments.

Detections now check read/write permissions before contacting ElasticSearch.

…ons_playbooks

`/detection/{id}?tab=[summary|comments|source|tuning|history]`

Also added the tab names to i18n.

…ons_playbooks

Query parameter can select ActiveTab

`/detection/{id}?tab=[summary|comments|source|tuning|history]`

Also added the tab names to i18n.

ElastAlert's Sigma rules can now be converted, viewed, and tested in Kibana.

Having a string value that needs to be parsed into an array is not the better choice over the config just using an array.

Tweak Wording

(Also includes CSS fixes for Detection DB details being better centered below the detail boxes.)

ElastAlert, Suricata, and Strelka now support `allowRegex` and `denyRegex` config options. These regexes are applied to community rules during the import process. The regex is applied to the entire rule body, one rule at a time. The rules are optional and case sensitive.

ElastAlert and Suricata regexes are applied before parsing the rules. Since Yara files tend to contain multiple rules, the rules must be parsed first and then filtered.

If an allowRegex is specified, the rule must match to be included. If a denyRegex is specified, the rule must NOT match to be included. If both regexes are specified and match, the rule will be denied. If both regexes are specified but do not match, the rule will be denied. Rules filtered out will not be stored in ElasticSearch or SOC.

Yara imports now are only including for the immediate following rule body and not all remaining rules in the file. Yara imports are considered part of the rule body when testing the regexes.

Changing any regex for any engine will not have an impact on already imported rules. However, existing rules may have their updates denied by these regexes. Due to the regexes being defined in soc.json, changing their values requires restarting the SOC service.

Updated tests.

A new config option allows strelka to leave yara rules uncompiled.

Apparently v-if on a v-slot doesn't make it conditional. Moved the v-if to the internal element achieves the intended results.

…import

Once a template is found, stop checking.

…to kilo